2026-10-29 00:47:21,758 [ 84937 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2026-10-29 00:47:21,759 [ 84937 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2026-10-29 00:47:21,759 [ 84937 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2026-10-29 00:47:21,759 [ 84937 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_smmkq4 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_aggregation_memory_efficient/test.py::test_remote 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True]' test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes test_backup_restore_on_cluster/test.py::test_empty_replicated_table test_backup_restore_on_cluster/test.py::test_file_deduplication test_backup_restore_on_cluster/test.py::test_get_error_from_other_host test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_backup_restore_on_cluster/test.py::test_mutation test_backup_restore_on_cluster/test.py::test_projection test_backup_restore_on_cluster/test.py::test_replicated_database test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False]' 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True]' test_backup_restore_on_cluster/test.py::test_system_functions test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile test_cgroup_limit/test.py::test_cgroup_cpu_limit 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node]' 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries]' test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd]' test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True]' test_cleanup_after_start/test.py::test_old_dirs_cleanup test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default test_config_xml_yaml_mix/test.py::test_extra_yaml_mix test_ddl_alter_query/test.py::test_alter test_ddl_alter_query/test.py::test_ddl_queue_hostname_change test_default_database_on_cluster/test.py::test_default_database_on_cluster test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 'test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed]' test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml test_disabled_access_control_improvements/test_row_policy.py::test_introspection test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies 'test_disk_over_web_server/test.py::test_cache[node2]' test_disk_over_web_server/test.py::test_incorrect_usage test_disk_over_web_server/test.py::test_replicated_database test_disk_over_web_server/test.py::test_unavailable_server 'test_disk_over_web_server/test.py::test_usage[node2]' test_disks_app_func/test.py::test_disks_app_func_cp test_disks_app_func/test.py::test_disks_app_func_ld test_disks_app_func/test.py::test_disks_app_func_ln test_disks_app_func/test.py::test_disks_app_func_ls test_disks_app_func/test.py::test_disks_app_func_mv test_disks_app_func/test.py::test_disks_app_func_read_write test_disks_app_func/test.py::test_disks_app_func_rm test_disks_app_func/test.py::test_remote_disk_list test_distributed_config/test.py::test_distibuted_settings test_distributed_ddl_password/test.py::test_alter test_distributed_ddl_password/test.py::test_truncate test_distributed_system_query/test.py::test_start_and_stop_replica_send test_enabling_access_management/test.py::test_enabling_access_management test_encrypted_disk/test.py::test_add_keys test_encrypted_disk/test.py::test_add_keys_with_id 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False]' 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True]' -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: xdist-3.5.0, random-0.2, reportlog-0.4.0, order-1.0.1, timeout-2.2.0, repeat-0.9.3 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_ddl_alter_query/test.py::test_alter test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] test_encrypted_disk/test.py::test_add_keys test_disk_over_web_server/test.py::test_cache[node2] test_disks_app_func/test.py::test_disks_app_func_cp test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed] test_buffer_profile/test.py::test_buffer_profile [gw8] [ 1%] PASSED test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible [gw9] [ 2%] PASSED test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile [gw8] [ 3%] PASSED test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible [gw9] [ 4%] PASSED test_buffer_profile/test.py::test_default_profile test_distributed_ddl_password/test.py::test_alter [gw2] [ 5%] PASSED test_disks_app_func/test.py::test_disks_app_func_cp test_disks_app_func/test.py::test_disks_app_func_ld test_aggregation_memory_efficient/test.py::test_remote [gw2] [ 6%] PASSED test_disks_app_func/test.py::test_disks_app_func_ld test_disks_app_func/test.py::test_disks_app_func_ln [gw2] [ 7%] PASSED test_disks_app_func/test.py::test_disks_app_func_ln test_disks_app_func/test.py::test_disks_app_func_ls [gw5] [ 8%] PASSED test_encrypted_disk/test.py::test_add_keys [gw2] [ 9%] PASSED test_disks_app_func/test.py::test_disks_app_func_ls test_disks_app_func/test.py::test_disks_app_func_mv test_encrypted_disk/test.py::test_add_keys_with_id [gw4] [ 10%] PASSED test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management [gw2] [ 11%] PASSED test_disks_app_func/test.py::test_disks_app_func_mv test_disks_app_func/test.py::test_disks_app_func_read_write [gw2] [ 12%] PASSED test_disks_app_func/test.py::test_disks_app_func_read_write test_disks_app_func/test.py::test_disks_app_func_rm [gw0] [ 13%] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] [gw1] [ 14%] PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] [gw2] [ 15%] PASSED test_disks_app_func/test.py::test_disks_app_func_rm test_disks_app_func/test.py::test_remote_disk_list test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] [gw2] [ 16%] PASSED test_disks_app_func/test.py::test_remote_disk_list [gw7] [ 17%] PASSED test_ddl_alter_query/test.py::test_alter test_ddl_alter_query/test.py::test_ddl_queue_hostname_change [gw4] [ 18%] PASSED test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management [gw5] [ 19%] PASSED test_encrypted_disk/test.py::test_add_keys_with_id test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml [gw4] [ 20%] PASSED test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml test_disabled_access_control_improvements/test_row_policy.py::test_introspection [gw5] [ 21%] PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] [gw4] [ 22%] PASSED test_disabled_access_control_improvements/test_row_policy.py::test_introspection [gw0] [ 23%] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] [gw9] [ 24%] PASSED test_aggregation_memory_efficient/test.py::test_remote test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies [gw5] [ 25%] PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] [gw1] [ 26%] PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries [gw7] [ 27%] PASSED test_ddl_alter_query/test.py::test_ddl_queue_hostname_change test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts [gw4] [ 28%] PASSED test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies [gw0] [ 29%] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica [gw8] [ 30%] PASSED test_distributed_ddl_password/test.py::test_alter test_distributed_ddl_password/test.py::test_truncate test_distributed_config/test.py::test_distibuted_settings test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default [gw1] [ 31%] FAILED test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works [gw0] [ 32%] PASSED test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async [gw1] [ 33%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes [gw1] [ 34%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] [gw8] [ 35%] PASSED test_distributed_ddl_password/test.py::test_truncate [gw4] [ 36%] PASSED test_distributed_config/test.py::test_distibuted_settings test_detached_parts_metrics/test.py::test_numbers_of_detached_parts [gw0] [ 37%] PASSED test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes [gw1] [ 38%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] test_backup_restore_on_cluster/test.py::test_empty_replicated_table [gw2] [ 39%] PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default [gw1] [ 40%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] test_cgroup_limit/test.py::test_cgroup_cpu_limit [gw6] [ 41%] PASSED test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed] test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed] test_enabling_access_management/test.py::test_enabling_access_management [gw0] [ 42%] PASSED test_backup_restore_on_cluster/test.py::test_empty_replicated_table [gw1] [ 43%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] test_backup_restore_on_cluster/test.py::test_file_deduplication test_config_xml_yaml_mix/test.py::test_extra_yaml_mix [gw1] [ 44%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] [gw8] [ 45%] PASSED test_cgroup_limit/test.py::test_cgroup_cpu_limit [gw1] [ 46%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] [gw0] [ 47%] PASSED test_backup_restore_on_cluster/test.py::test_file_deduplication [gw4] [ 48%] PASSED test_enabling_access_management/test.py::test_enabling_access_management test_backup_restore_on_cluster/test.py::test_get_error_from_other_host [gw5] [ 49%] PASSED test_detached_parts_metrics/test.py::test_numbers_of_detached_parts [gw0] [ 50%] PASSED test_backup_restore_on_cluster/test.py::test_get_error_from_other_host [gw7] [ 51%] PASSED test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async [gw1] [ 52%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_distributed_system_query/test.py::test_start_and_stop_replica_send [gw1] [ 53%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] [gw2] [ 54%] PASSED test_config_xml_yaml_mix/test.py::test_extra_yaml_mix [gw1] [ 55%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] [gw6] [ 56%] PASSED test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed] test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed] [gw0] [ 57%] PASSED test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_default_database_on_cluster/test.py::test_default_database_on_cluster [gw1] [ 58%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] test_backup_restore_on_cluster/test.py::test_mutation [gw3] [ 59%] PASSED test_disk_over_web_server/test.py::test_cache[node2] test_disk_over_web_server/test.py::test_incorrect_usage [gw1] [ 60%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] [gw5] [ 61%] PASSED test_distributed_system_query/test.py::test_start_and_stop_replica_send [gw3] [ 62%] PASSED test_disk_over_web_server/test.py::test_incorrect_usage test_disk_over_web_server/test.py::test_replicated_database [gw9] [ 63%] PASSED test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts [gw1] [ 64%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] [gw3] [ 65%] PASSED test_disk_over_web_server/test.py::test_replicated_database test_disk_over_web_server/test.py::test_unavailable_server [gw0] [ 66%] PASSED test_backup_restore_on_cluster/test.py::test_mutation [gw1] [ 67%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] test_backup_restore_on_cluster/test.py::test_projection test_cleanup_after_start/test.py::test_old_dirs_cleanup [gw1] [ 68%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] [gw0] [ 69%] PASSED test_backup_restore_on_cluster/test.py::test_projection [gw1] [ 70%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] test_backup_restore_on_cluster/test.py::test_replicated_database [gw7] [ 71%] PASSED test_default_database_on_cluster/test.py::test_default_database_on_cluster [gw1] [ 72%] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried [gw0] [ 73%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database [gw1] [ 74%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried test_backup_restore_on_cluster/test.py::test_replicated_database_async [gw9] [ 75%] PASSED test_cleanup_after_start/test.py::test_old_dirs_cleanup [gw1] [ 76%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] [gw0] [ 77%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts [gw0] [ 78%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts test_backup_restore_on_cluster/test.py::test_replicated_table [gw1] [ 79%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] [gw6] [ 80%] PASSED test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed] [gw0] [ 81%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster [gw0] [ 82%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster [gw3] [ 83%] PASSED test_disk_over_web_server/test.py::test_unavailable_server test_disk_over_web_server/test.py::test_usage[node2] [gw0] [ 84%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster [gw1] [ 85%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def [gw0] [ 86%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert [gw1] [ 87%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] [gw0] [ 88%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge [gw3] [ 89%] PASSED test_disk_over_web_server/test.py::test_usage[node2] [gw0] [ 90%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath [gw1] [ 91%] PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] [gw0] [ 92%] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges [gw0] [ 93%] PASSED test_backup_restore_on_cluster/test.py::test_required_privileges test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] [gw0] [ 94%] PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] [gw0] [ 95%] PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] test_backup_restore_on_cluster/test.py::test_system_functions [gw0] [ 96%] PASSED test_backup_restore_on_cluster/test.py::test_system_functions test_backup_restore_on_cluster/test.py::test_system_users [gw0] [ 97%] PASSED test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def [gw0] [ 98%] PASSED test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty [gw0] [ 99%] PASSED test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency [gw0] [100%] PASSED test_backup_restore_on_cluster/test.py::test_tables_dependency =================================== FAILURES =================================== ___________________ test_query_is_canceled_with_inf_retries ____________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 cluster = broken_s3 = def test_query_is_canceled_with_inf_retries(cluster, broken_s3): node = cluster.instances["node_with_inf_s3_retries"] broken_s3.setup_at_part_upload( count=10000000, after=2, action="connection_refused", ) insert_query_id = f"TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES" request = node.get_query_request( f""" INSERT INTO TABLE FUNCTION s3( 'http://resolver:8083/root/data/test_query_is_canceled_with_inf_retries', 'minio', 'minio123', 'CSV', auto, 'none' ) SELECT * FROM system.numbers LIMIT 1000000 SETTINGS s3_max_single_part_upload_size=100, s3_min_upload_part_size=10000, s3_check_objects_after_upload=0 """, query_id=insert_query_id, ) assert_eq_with_retry( node, f"SELECT count() FROM system.processes WHERE query_id='{insert_query_id}'", "1", ) assert_eq_with_retry( node, f"SELECT ProfileEvents['S3WriteRequestsErrors'] > 10 FROM system.processes WHERE query_id='{insert_query_id}'", "1", retry_count=12, sleep_time=10, ) > node.query(f"KILL QUERY WHERE query_id = '{insert_query_id}' ASYNC") test_checking_s3_blobs_paranoid/test.py:544: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3456: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 243, stderr: Received exception from server (version 24.3.18): E Code: 499. DB::Exception: Received from 172.16.7.6:9000. DB::Exception: Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 24.3.18.10427.altinitystable (altinity build)). Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d2e1b23 E 1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000eff24f4 E 2. ./contrib/llvm-project/libcxx/include/string:1499: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x0000000015e52f5d E 3. ./build_docker/./src/Common/ThreadPoolTaskTracker.cpp:118: void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000015a06669 E 4. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000015e0f700 E 5. ./contrib/llvm-project/libcxx/include/future:2069: std::packaged_task::operator()() @ 0x0000000011ee31d5 E 6. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000015e0fbed E 7. ./src/Common/Stopwatch.h:26: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000f0d6392 E 8. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f0db6fd E 9. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0db662 E 10. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl::worker(std::__list_iterator) @ 0x000000000f0d1e20 E 11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000f0d8612 E 12. __tsan_thread_start_func @ 0x000000000706b10f E 13. ? @ 0x00007f120cd28ac3 E 14. ? @ 0x00007f120cdba850 E . (S3_ERROR) E (query: KILL QUERY WHERE query_id = 'TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' ASYNC) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log setup ------------------------------ 2026-10-28 23:48:02 [ 514 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/mock_settings/reset'] (cluster.py:2046, exec_in_container) 2026-10-28 23:48:02 [ 514 ] DEBUG : Command:['docker', 'exec', 'roottestcheckings3blobsparanoid_resolver_1', 'curl', '-s', 'http://localhost:8083/mock_settings/reset'] (cluster.py:113, run_and_check) 2026-10-28 23:48:02 [ 514 ] DEBUG : Stdout:OK (cluster.py:121, run_and_check) ------------------------------ Captured log call ------------------------------- 2026-10-28 23:48:02 [ 514 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/mock_settings/at_part_upload?nothing=1&count=10000000&after=2&action=connection_refused'] (cluster.py:2046, exec_in_container) 2026-10-28 23:48:02 [ 514 ] DEBUG : Command:['docker', 'exec', 'roottestcheckings3blobsparanoid_resolver_1', 'curl', '-s', 'http://localhost:8083/mock_settings/at_part_upload?nothing=1&count=10000000&after=2&action=connection_refused'] (cluster.py:113, run_and_check) 2026-10-28 23:48:02 [ 514 ] DEBUG : Stdout:OK (cluster.py:121, run_and_check) 2026-10-28 23:48:02 [ 514 ] DEBUG : Executing query INSERT INTO TABLE FUNCTION s3( 'http://resolver:8083/root/data/test_query_is_canceled_with_inf_retries', 'minio', 'minio123', 'CSV', auto, 'none' ) SELECT * FROM system.numbers LIMIT 1000000 SETTINGS s3_max_single_part_upload_size=100, s3_min_upload_part_size=10000, s3_check_objects_after_upload=0 on node_with_inf_s3_retries (cluster.py:3520, get_query_request) 2026-10-28 23:48:02 [ 514 ] DEBUG : Executing query SELECT count() FROM system.processes WHERE query_id='TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' on node_with_inf_s3_retries (cluster.py:3455, query) 2026-10-28 23:48:02 [ 514 ] DEBUG : Executing query SELECT ProfileEvents['S3WriteRequestsErrors'] > 10 FROM system.processes WHERE query_id='TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' on node_with_inf_s3_retries (cluster.py:3455, query) 2026-10-28 23:48:13 [ 514 ] DEBUG : Executing query SELECT ProfileEvents['S3WriteRequestsErrors'] > 10 FROM system.processes WHERE query_id='TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' on node_with_inf_s3_retries (cluster.py:3455, query) 2026-10-28 23:48:13 [ 514 ] DEBUG : Executing query KILL QUERY WHERE query_id = 'TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' ASYNC on node_with_inf_s3_retries (cluster.py:3455, query) ============================== slowest durations =============================== 62.29s call test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed] 54.95s setup test_disk_over_web_server/test.py::test_cache[node2] 54.10s call test_disk_over_web_server/test.py::test_unavailable_server 53.80s call test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed] 47.91s call test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 44.67s call test_disk_over_web_server/test.py::test_cache[node2] 25.31s call test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed] 24.37s teardown test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] 24.16s teardown test_disks_app_func/test.py::test_remote_disk_list 24.12s call test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] 24.05s setup test_distributed_ddl_password/test.py::test_alter 22.64s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] 22.08s setup test_ddl_alter_query/test.py::test_alter 19.44s call test_disk_over_web_server/test.py::test_usage[node2] 18.53s call test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] 18.29s setup test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 17.61s call test_config_xml_yaml_mix/test.py::test_extra_yaml_mix 16.95s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 16.92s setup test_default_database_on_cluster/test.py::test_default_database_on_cluster 15.17s call test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 14.12s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] 14.08s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] 13.97s call test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 13.84s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] 13.70s setup test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 13.45s setup test_cleanup_after_start/test.py::test_old_dirs_cleanup 13.18s call test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] 13.11s call test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 13.06s call test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection 12.76s setup test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] 12.65s teardown test_ddl_alter_query/test.py::test_ddl_queue_hostname_change 11.82s call test_ddl_alter_query/test.py::test_ddl_queue_hostname_change 11.18s teardown test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 10.77s setup test_disks_app_func/test.py::test_disks_app_func_cp 10.73s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] 10.42s call test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] 10.30s call test_distributed_ddl_password/test.py::test_alter 10.01s call test_encrypted_disk/test.py::test_add_keys 10.01s setup test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default 9.88s teardown test_distributed_ddl_password/test.py::test_truncate 9.79s setup test_encrypted_disk/test.py::test_add_keys 9.67s setup test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 9.60s setup test_aggregation_memory_efficient/test.py::test_remote 9.36s teardown test_backup_restore_on_cluster/test.py::test_tables_dependency 9.28s call test_cgroup_limit/test.py::test_cgroup_cpu_limit 9.27s setup test_enabling_access_management/test.py::test_enabling_access_management 9.25s call test_aggregation_memory_efficient/test.py::test_remote 9.19s call test_cleanup_after_start/test.py::test_old_dirs_cleanup 9.17s call test_distributed_ddl_password/test.py::test_truncate 9.16s call test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 9.05s call test_backup_restore_on_cluster/test.py::test_replicated_database_async 8.97s teardown test_distributed_config/test.py::test_distibuted_settings 8.75s call test_encrypted_disk/test.py::test_add_keys_with_id 8.69s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 8.62s setup test_distributed_system_query/test.py::test_start_and_stop_replica_send 8.17s call test_backup_restore_on_cluster/test.py::test_mutation 7.91s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 7.85s call test_backup_restore_on_cluster/test.py::test_replicated_database 7.78s teardown test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default 7.77s teardown test_buffer_profile/test.py::test_default_profile 7.74s call test_backup_restore_on_cluster/test.py::test_file_deduplication 7.60s teardown test_distributed_system_query/test.py::test_start_and_stop_replica_send 7.43s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 7.35s call test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies 7.30s call test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management 7.29s call test_backup_restore_on_cluster/test.py::test_tables_dependency 7.19s teardown test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies 7.13s call test_backup_restore_on_cluster/test.py::test_required_privileges 7.11s setup test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection 6.99s teardown test_aggregation_memory_efficient/test.py::test_remote 6.88s call test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 6.84s setup test_distributed_config/test.py::test_distibuted_settings 6.78s teardown test_default_database_on_cluster/test.py::test_default_database_on_cluster 6.63s call test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 6.60s call test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 6.53s teardown test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 6.47s setup test_buffer_profile/test.py::test_buffer_profile 6.43s setup test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 6.33s call test_backup_restore_on_cluster/test.py::test_projection 6.31s setup test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed] 6.21s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 6.07s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] 6.00s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 5.93s call test_ddl_alter_query/test.py::test_alter 5.93s teardown test_disk_over_web_server/test.py::test_usage[node2] 5.75s teardown test_enabling_access_management/test.py::test_enabling_access_management 5.69s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 5.54s teardown test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed] 5.52s call test_backup_restore_on_cluster/test.py::test_empty_replicated_table 5.23s call test_backup_restore_on_cluster/test.py::test_replicated_table 5.20s call test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 5.18s teardown test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 4.98s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 4.87s teardown test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 4.81s call test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried 4.77s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] 4.72s call test_default_database_on_cluster/test.py::test_default_database_on_cluster 4.50s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 4.33s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] 4.31s call test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 4.30s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] 4.27s call test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 4.25s call test_disks_app_func/test.py::test_disks_app_func_cp 4.23s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] 4.17s call test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default 4.14s call test_backup_restore_on_cluster/test.py::test_system_users 4.09s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] 3.95s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] 3.93s call test_distributed_config/test.py::test_distibuted_settings 3.89s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] 3.86s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] 3.86s call test_backup_restore_on_cluster/test.py::test_system_functions 3.72s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] 3.65s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] 3.36s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] 3.36s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] 3.34s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] 3.33s teardown test_cleanup_after_start/test.py::test_old_dirs_cleanup 3.24s call test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] 3.21s call test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] 3.18s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] 3.16s call test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] 3.14s call test_disk_over_web_server/test.py::test_replicated_database 3.02s call test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 2.83s call test_disks_app_func/test.py::test_remote_disk_list 2.73s call test_distributed_system_query/test.py::test_start_and_stop_replica_send 2.67s call test_enabling_access_management/test.py::test_enabling_access_management 2.63s teardown test_backup_restore_on_cluster/test.py::test_empty_replicated_table 2.62s teardown test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 2.61s teardown test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 2.47s call test_disk_over_web_server/test.py::test_incorrect_usage 2.41s call test_disks_app_func/test.py::test_disks_app_func_rm 2.40s teardown test_backup_restore_on_cluster/test.py::test_file_deduplication 2.38s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 2.32s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 2.28s call test_disks_app_func/test.py::test_disks_app_func_ls 2.11s call test_disks_app_func/test.py::test_disks_app_func_ln 2.11s teardown test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 2.08s teardown test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 1.97s call test_disks_app_func/test.py::test_disks_app_func_mv 1.95s teardown test_backup_restore_on_cluster/test.py::test_mutation 1.86s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 1.84s call test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml 1.82s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 1.68s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_async 1.68s teardown test_backup_restore_on_cluster/test.py::test_projection 1.67s teardown test_backup_restore_on_cluster/test.py::test_replicated_database 1.67s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 1.62s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 1.62s teardown test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 1.62s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 1.61s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 1.57s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 1.57s teardown test_backup_restore_on_cluster/test.py::test_replicated_table 1.52s teardown test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 1.52s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 1.47s call test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 1.47s teardown test_backup_restore_on_cluster/test.py::test_system_functions 1.46s teardown test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] 1.46s teardown test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] 1.42s teardown test_backup_restore_on_cluster/test.py::test_system_users 1.39s teardown test_disabled_access_control_improvements/test_row_policy.py::test_introspection 1.31s teardown test_backup_restore_on_cluster/test.py::test_required_privileges 1.20s teardown test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management 1.14s call test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 1.04s teardown test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml 0.90s teardown test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection 0.69s call test_disks_app_func/test.py::test_disks_app_func_read_write 0.63s call test_buffer_profile/test.py::test_buffer_profile 0.59s call test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 0.58s call test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 0.53s call test_disabled_access_control_improvements/test_row_policy.py::test_introspection 0.53s call test_buffer_profile/test.py::test_default_profile 0.46s teardown test_encrypted_disk/test.py::test_add_keys_with_id 0.39s call test_disks_app_func/test.py::test_disks_app_func_ld 0.33s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] 0.28s teardown test_encrypted_disk/test.py::test_add_keys 0.27s teardown test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] 0.24s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] 0.22s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] 0.18s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] 0.16s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] 0.16s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] 0.16s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] 0.14s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] 0.13s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried 0.13s setup test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 0.12s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] 0.12s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] 0.12s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] 0.12s setup test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] 0.12s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] 0.12s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] 0.11s setup test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 0.10s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 0.09s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] 0.08s setup test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] 0.07s setup test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] 0.01s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table 0.00s teardown test_disk_over_web_server/test.py::test_cache[node2] 0.00s setup test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 0.00s teardown test_ddl_alter_query/test.py::test_alter 0.00s setup test_cgroup_limit/test.py::test_cgroup_cpu_limit 0.00s teardown test_disks_app_func/test.py::test_disks_app_func_cp 0.00s setup test_config_xml_yaml_mix/test.py::test_extra_yaml_mix 0.00s teardown test_distributed_ddl_password/test.py::test_alter 0.00s setup test_disk_over_web_server/test.py::test_replicated_database 0.00s setup test_backup_restore_on_cluster/test.py::test_file_deduplication 0.00s setup test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] 0.00s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 0.00s setup test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 0.00s teardown test_config_xml_yaml_mix/test.py::test_extra_yaml_mix 0.00s setup test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] 0.00s setup test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 0.00s setup test_backup_restore_on_cluster/test.py::test_tables_dependency 0.00s setup test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies 0.00s setup test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 0.00s setup test_backup_restore_on_cluster/test.py::test_empty_replicated_table 0.00s setup test_backup_restore_on_cluster/test.py::test_projection 0.00s setup test_backup_restore_on_cluster/test.py::test_mutation 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] 0.00s setup test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 0.00s setup test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml 0.00s setup test_disabled_access_control_improvements/test_row_policy.py::test_introspection 0.00s setup test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 0.00s setup test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] 0.00s setup test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed] 0.00s setup test_disk_over_web_server/test.py::test_usage[node2] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] 0.00s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 0.00s teardown test_disks_app_func/test.py::test_disks_app_func_ln 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 0.00s setup test_backup_restore_on_cluster/test.py::test_required_privileges 0.00s setup test_disk_over_web_server/test.py::test_incorrect_usage 0.00s setup test_buffer_profile/test.py::test_default_profile 0.00s setup test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] 0.00s teardown test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 0.00s setup test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 0.00s teardown test_cgroup_limit/test.py::test_cgroup_cpu_limit 0.00s setup test_disks_app_func/test.py::test_remote_disk_list 0.00s setup test_disks_app_func/test.py::test_disks_app_func_ls 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] 0.00s setup test_ddl_alter_query/test.py::test_ddl_queue_hostname_change 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database 0.00s setup test_disks_app_func/test.py::test_disks_app_func_ld 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_async 0.00s setup test_disks_app_func/test.py::test_disks_app_func_read_write 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 0.00s setup test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] 0.00s teardown test_disks_app_func/test.py::test_disks_app_func_rm 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] 0.00s teardown test_buffer_profile/test.py::test_buffer_profile 0.00s setup test_backup_restore_on_cluster/test.py::test_system_functions 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 0.00s setup test_disks_app_func/test.py::test_disks_app_func_rm 0.00s teardown test_disks_app_func/test.py::test_disks_app_func_mv 0.00s setup test_encrypted_disk/test.py::test_add_keys_with_id 0.00s setup test_distributed_ddl_password/test.py::test_truncate 0.00s teardown test_disks_app_func/test.py::test_disks_app_func_read_write 0.00s setup test_backup_restore_on_cluster/test.py::test_system_users 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried 0.00s teardown test_disk_over_web_server/test.py::test_unavailable_server 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] 0.00s setup test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] 0.00s setup test_disk_over_web_server/test.py::test_unavailable_server 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] 0.00s setup test_disks_app_func/test.py::test_disks_app_func_ln 0.00s teardown test_disk_over_web_server/test.py::test_incorrect_usage 0.00s teardown test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] 0.00s teardown test_disk_over_web_server/test.py::test_replicated_database 0.00s setup test_disks_app_func/test.py::test_disks_app_func_mv 0.00s teardown test_disks_app_func/test.py::test_disks_app_func_ls 0.00s teardown test_disks_app_func/test.py::test_disks_app_func_ld =========================== short test summary info ============================ FAILED test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries PASSED test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible PASSED test_buffer_profile/test.py::test_buffer_profile PASSED test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible PASSED test_buffer_profile/test.py::test_default_profile PASSED test_disks_app_func/test.py::test_disks_app_func_cp PASSED test_disks_app_func/test.py::test_disks_app_func_ld PASSED test_disks_app_func/test.py::test_disks_app_func_ln PASSED test_encrypted_disk/test.py::test_add_keys PASSED test_disks_app_func/test.py::test_disks_app_func_ls PASSED test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection PASSED test_disks_app_func/test.py::test_disks_app_func_mv PASSED test_disks_app_func/test.py::test_disks_app_func_read_write PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node] PASSED test_disks_app_func/test.py::test_disks_app_func_rm PASSED test_disks_app_func/test.py::test_remote_disk_list PASSED test_ddl_alter_query/test.py::test_alter PASSED test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management PASSED test_encrypted_disk/test.py::test_add_keys_with_id PASSED test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] PASSED test_disabled_access_control_improvements/test_row_policy.py::test_introspection PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] PASSED test_aggregation_memory_efficient/test.py::test_remote PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] PASSED test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries] PASSED test_ddl_alter_query/test.py::test_ddl_queue_hostname_change PASSED test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] PASSED test_distributed_ddl_password/test.py::test_alter PASSED test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br] PASSED test_distributed_ddl_password/test.py::test_truncate PASSED test_distributed_config/test.py::test_distibuted_settings PASSED test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2] PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate] PASSED test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed] PASSED test_backup_restore_on_cluster/test.py::test_empty_replicated_table PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4] PASSED test_cgroup_limit/test.py::test_cgroup_cpu_limit PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none] PASSED test_backup_restore_on_cluster/test.py::test_file_deduplication PASSED test_enabling_access_management/test.py::test_enabling_access_management PASSED test_detached_parts_metrics/test.py::test_numbers_of_detached_parts PASSED test_backup_restore_on_cluster/test.py::test_get_error_from_other_host PASSED test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd] PASSED test_config_xml_yaml_mix/test.py::test_extra_yaml_mix PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br] PASSED test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed] PASSED test_backup_restore_on_cluster/test.py::test_keeper_value_max_size PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2] PASSED test_disk_over_web_server/test.py::test_cache[node2] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate] PASSED test_distributed_system_query/test.py::test_start_and_stop_replica_send PASSED test_disk_over_web_server/test.py::test_incorrect_usage PASSED test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip] PASSED test_disk_over_web_server/test.py::test_replicated_database PASSED test_backup_restore_on_cluster/test.py::test_mutation PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4] PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none] PASSED test_backup_restore_on_cluster/test.py::test_projection PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz] PASSED test_default_database_on_cluster/test.py::test_default_database_on_cluster PASSED test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried PASSED test_cleanup_after_start/test.py::test_old_dirs_cleanup PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_async PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False] PASSED test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster PASSED test_disk_over_web_server/test.py::test_unavailable_server PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert PASSED test_disk_over_web_server/test.py::test_usage[node2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge PASSED test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath PASSED test_backup_restore_on_cluster/test.py::test_required_privileges PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False] PASSED test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True] PASSED test_backup_restore_on_cluster/test.py::test_system_functions PASSED test_backup_restore_on_cluster/test.py::test_system_users PASSED test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def PASSED test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty PASSED test_backup_restore_on_cluster/test.py::test_tables_dependency =================== 1 failed, 99 passed in 284.87s (0:04:44) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 437, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_smmkq4 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_aggregation_memory_efficient/test.py::test_remote 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True]' test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes test_backup_restore_on_cluster/test.py::test_empty_replicated_table test_backup_restore_on_cluster/test.py::test_file_deduplication test_backup_restore_on_cluster/test.py::test_get_error_from_other_host test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_backup_restore_on_cluster/test.py::test_mutation test_backup_restore_on_cluster/test.py::test_projection test_backup_restore_on_cluster/test.py::test_replicated_database test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[False]' 'test_backup_restore_on_cluster/test.py::test_stop_other_host_during_backup[True]' test_backup_restore_on_cluster/test.py::test_system_functions test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile test_cgroup_limit/test.py::test_cgroup_cpu_limit 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node]' 'test_checking_s3_blobs_paranoid/test.py::test_adaptive_timeouts[node_with_inf_s3_retries]' test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries test_checking_s3_blobs_paranoid/test.py::test_upload_after_check_works 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_create_multi_part_upload[zstd]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[br]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[bz2]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[deflate]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[gzip]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[lz4]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[none]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[xz]' 'test_checking_s3_blobs_paranoid/test.py::test_upload_s3_fail_upload_part_when_multi_part_upload[zstd]' test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_refused_is_retried 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_create_mpu_retried[True]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[False]' 'test_checking_s3_blobs_paranoid/test.py::test_when_s3_connection_reset_by_peer_at_upload_is_retried[True]' test_cleanup_after_start/test.py::test_old_dirs_cleanup test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default test_config_xml_yaml_mix/test.py::test_extra_yaml_mix test_ddl_alter_query/test.py::test_alter test_ddl_alter_query/test.py::test_ddl_queue_hostname_change test_default_database_on_cluster/test.py::test_default_database_on_cluster test_detached_parts_metrics/test.py::test_numbers_of_detached_parts 'test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_complex[complex_key_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_ranged[range_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_executable_hashed.py::test_simple[hashed]' test_disabled_access_control_improvements/test_row_policy.py::test_dcl_introspection test_disabled_access_control_improvements/test_row_policy.py::test_dcl_management test_disabled_access_control_improvements/test_row_policy.py::test_dcl_users_with_policies_from_users_xml test_disabled_access_control_improvements/test_row_policy.py::test_introspection test_disabled_access_control_improvements/test_row_policy.py::test_some_users_without_policies 'test_disk_over_web_server/test.py::test_cache[node2]' test_disk_over_web_server/test.py::test_incorrect_usage test_disk_over_web_server/test.py::test_replicated_database test_disk_over_web_server/test.py::test_unavailable_server 'test_disk_over_web_server/test.py::test_usage[node2]' test_disks_app_func/test.py::test_disks_app_func_cp test_disks_app_func/test.py::test_disks_app_func_ld test_disks_app_func/test.py::test_disks_app_func_ln test_disks_app_func/test.py::test_disks_app_func_ls test_disks_app_func/test.py::test_disks_app_func_mv test_disks_app_func/test.py::test_disks_app_func_read_write test_disks_app_func/test.py::test_disks_app_func_rm test_disks_app_func/test.py::test_remote_disk_list test_distributed_config/test.py::test_distibuted_settings test_distributed_ddl_password/test.py::test_alter test_distributed_ddl_password/test.py::test_truncate test_distributed_system_query/test.py::test_start_and_stop_replica_send test_enabling_access_management/test.py::test_enabling_access_management test_encrypted_disk/test.py::test_add_keys test_encrypted_disk/test.py::test_add_keys_with_id 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False]' 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True]' -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 ' returned non-zero exit status 1.